“If you spend a little time with CDC’s interactive maps, the conclusion seems blindingly obvious: measures taken by the states to “fight” the Wuhan virus, something that was never deemed possible before last year, imposed enormous costs but have made little or no difference in the end. The smart states were the ones the press hated, that recommended common-sense measures to protect those most at risk, but didn’t impose catastrophic damages on their citizens in a futile effort to stop a virus from spreading through the population”

 says John Hinderaker of Powerline Blog in  “Fighting COVID? It Just Didn’t Matter.”

That is an example of a flawed argument where the conclusion – that lockdowns had little effect in stopping the spread of COVID-19 – is nevertheless likely true. But it is still worth spelling out this type of error a little further because one sees it so often in reasoning about cause and effect.

The argument is that while some American states adopted strict lockdowns, others did not, and yet they all ended with similar COVID outcomes. Therefore lockdowns did not affect COVID.

But suppose COVID hit some states harder than others because of population demographics, climate, or other unknown factors.  Then harder-hit states would likely impose more intense lockdowns than places with a less severe outbreak. More intense lockdowns could have suppressed the infection so effectively that COVID outcomes in the harder-hit states ended up roughly the same as in places that were less hard-hit, with less forceful lockdowns. One might mistakenly conclude that lockdowns did not work even though they had been highly effective.

Another example of this error would be the argument that hiring more cops does not reduce crime since there tends to be, if anything, a positive correlation between crime rates and policing levels across cities.   However, some cities are more crime-prone than others to start with (because of factors such as demographics or the local economy) and will hire more police for that very reason.  In reality, more police could have a powerful effect in reducing crime to less than what it would have been otherwise, but you would not see this from the simple correlation of crime rates and police levels across cities.

So, to the well-known rule that “Correlation does not imply Causation,” one could add “Lack of Correlation does not imply Lack of Causation.”

The underlying problem is the same in both examples.  To figure out the actual effects of differences in lockdown or police policies, one needs to compare across states or cities that are otherwise more or less similar. But that assumption may not be satisfied.  In the first example, states might differ by the severity of the COVID outbreak, which then influences the severity of the chosen lockdown policy. In the second, cities differ because of their underlying propensity for crime, which affects the level of policing selected. Hence the aptness of the technical name for this problem: “selection bias.”

Fortunately, there are statistical methods to grapple with selection bias. Studies using these methods suggest that differences in the severity of lockdowns did not, in fact, have much of a causal effect on COVID outcomes. So, despite a flawed argument, Mr. Hinderaker’s conclusion was correct. On the other hand, numerous studies applying these methods conclude that more policing does have a strong causal effect in reducing crime. See, for example, the literature survey in this 2012 DOJ study on “The Relationship between Economic Conditions, Policing and Crime Trends.”